-
Notifications
You must be signed in to change notification settings - Fork 5.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wip improved object retrieval #10513
Conversation
llama_index/core/base_retriever.py
Outdated
@@ -144,7 +167,7 @@ def _handle_recursive_retrieval( | |||
node = n.node | |||
score = n.score or 1.0 | |||
if isinstance(node, IndexNode): | |||
obj = self.object_map.get(node.index_id, None) | |||
obj = node.obj or self.object_map.get(node.index_id, None) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
help me understand, is object_map
now only used for retrievers/query engines?
If it's a Node it should now be serialized/deserialized directly on the IndexNode
right?
at a high-level once we make retrievers/query engine serializable i was thinking object_map would go away, and we'd replace with a proper docstore
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes if query engines/retrievers were serializable, this would go away.
Right now, unserializable index nodes have to be passed in under the objects
kwarg -- from there, we can build a map of index id to object
Then we can serialize and retrieve the index node without the object.
If an index node is retrieved, the object map is checked if we have its object
nice will upd some notebooks! |
A WIP approach to improve recurisve retrieval.
Basically, if
index_node.obj
is serializable, then just throw it into the vector db/storage layer, no need for mappingsExample Usage
TODO